Skip to content

[SPARK-23914][SQL][follow-up] refactor ArrayUnion#21937

Closed
kiszk wants to merge 4 commits intoapache:masterfrom
kiszk:SPARK-23914-follow
Closed

[SPARK-23914][SQL][follow-up] refactor ArrayUnion#21937
kiszk wants to merge 4 commits intoapache:masterfrom
kiszk:SPARK-23914-follow

Conversation

@kiszk
Copy link
Member

@kiszk kiszk commented Jul 31, 2018

What changes were proposed in this pull request?

This PR refactors ArrayUnion based on this suggestion.

  1. Generate optimized code for all of the primitive types except boolean
  2. Generate code using ArrayBuilder or ArrayBuffer
  3. Leave only a generic path in the interpreted path

How was this patch tested?

Existing tests

@holdensmagicalunicorn
Copy link

holdensmagicalunicorn commented Jul 31, 2018

@kiszk, thanks! I am a bot who has found some folks who might be able to help with the review:@ueshin, @rxin and @colorant

@SparkQA
Copy link

SparkQA commented Jul 31, 2018

Test build #93840 has finished for PR 21937 at commit 738a4a0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class OpenHashSet[@specialized(Long, Int, Double, Float) T: ClassTag](
  • sealed class Hasher[@specialized(Long, Int, Double, Float) T] extends Serializable
  • class DoubleHasher extends Hasher[Double]
  • class FloatHasher extends Hasher[Float]
  • case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike

@kiszk kiszk force-pushed the SPARK-23914-follow branch from 738a4a0 to f48dde0 Compare August 4, 2018 07:28
@SparkQA
Copy link

SparkQA commented Aug 4, 2018

Test build #94208 has finished for PR 21937 at commit f48dde0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

ueshin
ueshin reviewed Aug 6, 2018
View reviewed changes
...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala Outdated
Copy link
Member

@ueshin ueshin Aug 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: new $arrayBuilderClass() should work?

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala Outdated
Copy link
Member

@ueshin ueshin Aug 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: classOf[mutable.ArrayBuilder[_]].getName?

@ueshin
Copy link
Member

ueshin commented Aug 6, 2018

I'll revisit after the conflict is fixed.

@kiszk
Copy link
Member Author

kiszk commented Aug 6, 2018

I see. Now, I am rebasing and resolving conflicts.

@kiszk kiszk force-pushed the SPARK-23914-follow branch from f48dde0 to 93ec9ec Compare August 6, 2018 15:35
@kiszk kiszk changed the title [WIP][SPARK-23914][SQL][follow-up] refactor ArrayUnion [SPARK-23914][SQL][follow-up] refactor ArrayUnion Aug 6, 2018
ueshin
ueshin reviewed Aug 6, 2018
View reviewed changes
...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
val classTag = s"scala.reflect.ClassTag$$.MODULE$$.$hsTypeName()"
val hashSet = ctx.freshName("hashSet")
val arrayBuilder = classOf[mutable.ArrayBuilder[_]].getName
val arrayBuilderClass = s"$arrayBuilder$$of$ptName"
Copy link
Member

@ueshin ueshin Aug 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not needed any more?

Copy link
Member Author

@kiszk kiszk Aug 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we still need this to create an intermediate result array. The array is allocated at L3907 as other functions do.

Copy link
Member

@ueshin ueshin Aug 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry, I misread.

@kiszk kiszk mentioned this pull request Aug 6, 2018
Closed
@SparkQA
Copy link

SparkQA commented Aug 6, 2018

Test build #94290 has finished for PR 21937 at commit 93ec9ec.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 6, 2018

Test build #94304 has finished for PR 21937 at commit 26c1f8a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@ueshin
Copy link
Member

ueshin commented Aug 7, 2018

LGTM.

@ueshin
Copy link
Member

ueshin commented Aug 7, 2018

Thanks! merging to master.

@asfgit asfgit closed this in 4446a0b Aug 7, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ueshin ueshin ueshin left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kiszk @holdensmagicalunicorn @SparkQA @ueshin